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(54) SIGNAL PROCESSING DEVICE AND METHOD THEREFOR AND PROGRAM STORING 
MEDIUM 



(57) In a signal processing device and its method as 
well as a program storing medium, psychological audi- 
tory sense analysis of the survival state of an input signal 
of the time when the input signal has been compressed 
is performed in response to the compression process- 
ing, and the result of the very analysis is outputted as 
the psychological auditory sense encoded information, 



and digital watermarking information is superimposed 
on the input signal on the basis of the psychological au- 
ditory sense encoded information, so that it is possible 
to easily realize digital watermarking information that 
has large compression resistance and digital water- 
marking information that has small compression resist- 
ance. 
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Description 

Technical Field 

[0001] This invention is directed to a signal process- 
ing device and its method as well as a program storing 
medium, and suitably applicable to the safeguarding of 
the copyright of music data and so on. 

Background Art 

[0002] In recentyears, music contents and so on have 
been distributed to an indefinite number of network ter- 
minal devices (computers) from a Web site via a network 
such as the Internet. 

[0003] A computer user is able to record the distribut- 
ed music contents on a wide variety of recording media 
in a state of digital data, so as to generate a music soft 
of favorite music that is recorded with a satisfactory tone 
quality, as the user wishes. 

[0004] In this way, a computer user has accessed a 
Web site and recorded music contents on a recording 
medium freely. However, the act of distributing music 
contents by the person who set up the Web site without 
permission of the copyrighter, who is the owner of the 
music contents, will cause a problem of an infringement 
of the copyright of the music contents. 

Disclosure of Invention 

[0005] Considering aforementioned respects, this in- 
vention proposes a signal processing device and its 
method as well as a program storing medium that can 
effectively suppress the infringement of the copyright of 
contents that have been distributed to an indefinite 
number of terminal devices via, for instance, a network 
[0006] In orderto solve such a problem, in the present 
invention, psychological auditory sense analysis of the 
survival state of an input signal of the time when the in- 
put signal has been compressed is performed in re- 
sponse to the compression processing, and the result 
of the very analysis is outputted as the 
psychological auditory sense encoded information, and 
digital watermarking information is superimposed on the 
input signal on the basis of the psychological auditory 
sense encoded information, so that it is possible to eas- 
ily generate digital watermarking information that has 
large compression resistance and digital watermarking 
information that has small compression resistance. 

Brief Description of Drawings 

[0007] 

Fig. 1 is a block diagram illustrating a general con- 
figuration of a copy control system in which a wa- 
termark is utilized according to the present inven- 
tion. 



Fig. 2 is a schematic diagram showing an example 
of application of the copy controlling in the copy con- 
trol system. 

Fig. 3 is a block diagram illustrating a configuration 
of the watermark encoder according to an embodi- 
ment of the present invention. 
Fig. 4 is a schematic diagram used for explaining 
the principle of superimposing a watermark. 
Fig. 5 is a schematic diagram used for explaining 
the processing of the psychological auditory sense 
analysis division. 

Fig. 6 is a schematic diagram used for explaining 
the processing of the psychological auditory sense 
analysis division. 

Fig. 7 is a schematic diagram used for explaining 
the psychological auditory sense encoding. 
Fig. 8 is a schematic diagram showing an example 
of encoding of the watermarks according to the 
present invention. 

Fig. 9 is a flow chart showing a procedure of the 
watermark encoder. 

Fig. 1 0 is a block diagram illustrating a configuration 
of the watermark decoder. 

Fig. 11 is a flow chart showing a procedure of the 
watermark decoder. 

Fig. 12 is a block diagram illustrating an internal 
configuration of a computer. 

Best Mode for Carrying out the Invention 

[0008] An embodiment of the present invention will be 
described in detail hereinafter with reference to the 
drawings. 

[0009] In Fig. 1,10 shows a contents copy control sys- 
tem as a whole; at the time when it records a digital audio 
signal DA, which is the audio source, on an optical disk 
2, it adds a robust (strong) watermark DC1 and a fragile 
(weak) watermark DC2 to the digital audio signal DA and 
then records them on the optical disk 2, by means of a 
watermark (digital watermarking information) encoder 
1. 

[0010] At here, the robust watermark DC1 is com- 
prised of information of the audio source, such as the 
name of the copyright holder, and whether the copying 
is permitted or not; the fragile watermark DC2 is com- 
prised of information such as a trigger for detecting the 
compression history. In this connection, the robust wa- 
termark DC1 is a watermark that does not vanish even 
if it has passed through a compressing system of MP3 
and so on, and the fragile watermark DC2 is a water- 
mark that vanishes (the survival rate becomes low) 
when it has passed through a compressing system of 
MP3 and so on, because.of its quantization error. 
[0011] In this way, in the case where a compressed 
stream signal DS, which has been uploaded to a site on 
a network 4 via a compression processing by, for in- 
stance, a MP3 (MPEG Audio Layer 3) encoder 3 and so 
on, out of optical disks 2 on which the digital audio signal 
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DA including the robust watermark DC1 and the fragile 
watermark DC2 has been recorded, is downloaded to a 
personal computer 5, and also in the case where an op- 
tical disk 2 that is distributed without being compressed, 
out of optical disks 2 on which the digital audio signal 
DA including the robust watermark DC1 and the fragile 
watermark DC2 has been recorded, is loaded to the per- 
sonal computer 5, the personal computer 5 first judges 
based on a signal that is input to a Codec (Code and 
Decode) comprising a modem (Modulator Demodula- 
tor) whether the input signal to the personal computer 5 
is the non-compressed digital audio signal DA or the 
compressed stream signal DS, and by what compres- 
sion standard the compressed stream signal DS has 
been encoded and then selectively switches a switch 
SW according to the judgement result. 
[001 2] And, the personal computer 5 detects the em- 
bedded copyright information out of the input signal of 
the personal computer 5 by means of a watermark de- 
coder 6, and controls copying of the input signal (the 
digital audio signal DA, the compressed stream signal 
DS and so on) to a PD (portable device) 8 such as a 
memory card. That is, the watermark decoder 6 detects 
the robust watermark DC1 and the fragile watermark 
DC2. A LCM (licensed SDMI (secure digital music initi- 
ative) compliant module) 7 controls whether the music 
contents are copied to the PD 8 or not, on the basis of 
two kinds of watermarks (the robust watermark DC1 and 
the fragile watermark DC2) that have been detected by 
the watermark decoder 6. 

[0013] Fig. 2 shows an example of controlling of 
whether the music contents are copied to the PD 8 or 
not, which is performed by the LCM 7, using the detect- 
ed two kinds of watermarks; in the case where the robust 
watermark DC1 has been detected and the fragile wa- 
termark DC2 has not been detected, the LCM 7 deter- 
mines that the music contents have some compression 
histories, and inhibits the PD 8 from copying the very 
music contents. 

[0014] In the case where both the robust watermark 
DC1 and the fragile watermark DC2 are detected, the 
LCM 7 determines that no compression history could be 
found and permits copying of the very music contents 
to the PD 8. 

[001 5] Furthermore, in the case where both the robust 
watermark DC1 and the fragile watermark DC2 are not 
detected, the LCM 7 permits copying of the very music 
contents to the PD 8 because of no information as to the 
copyrighter of the music content or permission of copy- 
ing. There is no technical possibility of a case where the 
robust watermark DC1 is not detected and the fragile 
watermark DC2 is either detected, which is thus excep- 
tional. 

[001 6] Fig. 3 shows a configuration of the watermark 
encoder 1 ; a psychological auditory sense analysis di- 
vision 1 1 performs the analysis of the sequentially input- 
ted digital audio signal DA1 , utilizing a psychological au- 
ditory sense model, and controls a watermark superim- 



posing division 16 that has a robust watermark super- 
imposing division and a fragile watermark superimpos- 
ing division. The digital audio signal DA1 generates a 
MDCT coefficient D1 4 at a MDCT (modified discrete co- 
5 sine transform) processing division 1 4, and then outputs 
this to the watermark superimposing process division 
16. 

[0017] The watermark superimposing division 16 is 
adapted to respectively superimposing the robust wa- 

10 termark DC 1 and the fragile watermark DC2 on a spec- 
trum of the frequency band that has been determined 
on the basis of the analysis result of the psychological 
auditory sense analysis division 11 mentioned below, 
out of a frequency spectrum of time series sample 

'5 blocks of the digital audio signal DA1 that has been MD- 
CT converted. 

[0018] More concretely, the watermark superimpos- 
ing division 16 extracts a spectrum of the frequency 
band f1 on which to superimpose the robust watermark 
20 DC1 and the fragile watermark DC2, out of the frequen- 
cy spectrum of time series sample blocks of the input 
audio signal DA1 that has been MDCT converted (Fig. 
4(A)). 

[001 9] The watermark superimposing division 1 6 then 
25 scales the frequency spectrum f 1 at a stated rate, and 
generates a frequency spectrum f2 that. is attenuated 
(Fig. 4(B)). If the polarity of data to be embedded is "1", 
the frequency spectrum f2 is used as it is. If the polarity 
of data to be embedded is "O", the frequency spectrum 
30 f2 with the positive and negative inverted is used. 

[0020] The watermark superimposing division 16 
shifts the frequency spectrum f2 in the direction of higher 
or lower frequency to keep a distance for four bars of 
spectrum (Fig. 4(C)). In Fig. 4C, the frequency spectrum 
35 f2 is shifted for four bars of spectrum in the direction of 
higher frequency to generate a frequency spectrum f3. 
[0021] Next, the watermark superimposing division 
16 superimposes the robust watermark DC1 and the 
fragile watermark DC2 on a frequency spectrum f4 (Fig. 
40 4 (D)), which has been obtained by adding the shifted 
frequency spectrum f3 (Fig. 4 (C)) to the original fre- 
quency spectrum f 1 (Fig. 4(A)). 
[0022] At the point of addition, the correlation between 
the polarities of a certain frequency spectrum and a fre- 
45 quency spectrum that is away by four bars is processed 
to be inclined to the positive or negative. This processing 
is performed only when the sign of the frequency spec- 
trum f4 is inverted by adding the frequency spectrum f3 
to the frequency spectrum f1 , but not performed at other 
50 times when the sign is not inverted. In Fig. 4(D), blank 
frames represent signal elements eliminated from the 
frequency spectrum f1 (Fig. 4(A)) and hatching frames 
represent signal elements obtained from the addition. 
[0023] This can reduce the amount of embedded wa- 
55 termarks which causes deterioration of tone quality and 
does not contribute to the accuracy of detection. More- 
over, by scaling the original frequency spectrum and 
adding it to its adjacency, embedding with reduced de- 
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terioration of tone quality will be possible using the 
masking effect. 

[0024] In this connection, if it adds the level that has 
been obtained by attenuating the level of the first fre- 
quency component f 1 (Fig. 4(A)) in which the watermark 
is to be embedded in a definite rate and that the polarity 
of the watermark to be embedded is multiplied, to the 
level of the second frequency component f3 (Fig. 4(C)) 
that is away from the first frequency component f2 (Fig. 
4(B)) by, for example, 4 frequency components, and em- 
beds the watermark, then, after the watermark has been 
embedded, such a probability is heightened that the po- 
larity of the correlation between the first frequency com- 
ponentfl (Fig. 4(A)) and the level of the second frequen- 
cy component f3 (Fig. 4(C)) becomes the same (positive 
or negative), and this is utilized for detection of the wa- 
termark in a watermark decoder that will be mentioned 
below. 

[0025] Inthisway, on the basis of the result D11 ofthe 
analysis performed in the psychological auditory sense 
analysis division 1 1 , the watermark superimposing proc- 
ess division 16 embeds the robust watermark DC1 and 
the fragile watermark DC2, so as to become the fre- 
quency band and the sound pressure level that include 
the consideration for the threshold of audibility and/or 
the masking threshold effect mentioned below, in the 
previously and separately allocated plural frequency 
bands, respectively. 

[0026] That is, the psychological auditory sense anal- 
ysis division 11 analyzes the input digital audio signal 
DA1 , and calculates from curves of original sound and 
threshold of audibility a masking threshold which is the 
limit to being able to perceive actually listening to the 
original sound. The masking capability toward the given 
digital audio signal DA1 depends on the frequency band 
and the level of sound pressure. 
[0027] Generally speaking, in an encoder for com- 
pression processing, on the basis of the information of 
the frequency band and the level of sound pressure, the 
best method for representing the input audio signal with 
the restricted bit resolution would be determined. That 
is, by utilizing a psychological auditory sense model that 
is used for compression, and embedding the watermark 
with a position and an energy of the frequency that tends 
to be eliminated by the compression process, it is pos- 
sible to realize the fragile watermark DC2. 
[0028] Hereinafter, the principle of the method for em- 
bedding the robust watermark DC1, which is resistant 
to compression, and the fragile watermark DC2, which 
is easily affected by compression, utilizing the frequency 
band that tends to be eliminated by compression proc- 
ess is explained. 

[0029] Fig. 5 shows a threshold of audibility; a thresh- 
old of audibility of silent time means the minimum level 
of sound that can be detected by the auditory sense, 
and relates to a boundary of a noise that can be per- 
ceived by the auditory sense at a silent time. As shown 
in Fig. 5, a sound whose sound pressure level is higher 



than the threshold of audibility of silent time like the tone 
A can be perceived, but a sound whose sound pressure 
level is lower than the threshold of audibility of silent time 
like the tone B can not be perceived. Moreover, as 
5 shown in Fig. 5, a threshold of audibility of silent time 
depends on frequency, and even if the tones A and B 
have the same sound pressure level, they can be per- 
ceived or not depending upon the frequency of the 
sound. 

w [0030] And, Fig. 6 shows a masking effect; a masking 
effect mainly relates to the detection threshold of quan- 
tization distortion and/or background noise, and the de- 
tection threshold of the specific sound greatly varies de- 
pending on the other sounds that are heard at the same 

'5 time. As shown in Fig. 6, as to a definite frequency range 
concerning thetoneC, the other sounds become difficult 
to be perceived; for instance, the tone D is almost inau- 
dible in spite of the fact that it is a pure tone that has 
comparatively high sound pressure, on the other hand, 

20 the tone E is audible. The nearer the frequency of the 
masker (a sound that masks the other) and the frequen- 
cy of the maskee (a sound that is masked and made 
inaudible) are positioned, the stronger the masking ef- 
fect operates. 

25 [0031] Therefore, in the watermark encoder 1 , utiliz- 
ing these principles, the process of embedding the wa- 
termarks (the robust watermark DC1 and the fragile wa- 
termark DC2) is achieved wherein the compression 
characteristics of MP3 is considered. 

30 [0032] In this connection, Fig. 7 shows encoding (psy- 
chological auditory sense encoding in which psycholog- 
ical auditory sense is considered) performed in a MP3 
encoder 3 (Fig. 1); for instance, after the MP3 encoder 
3 has divided the input audio signal DA into 32 frequency 

35 bands (subbands), it performs psychological auditory 
sense encoding in such a manner that the quantization 
noise falls under the threshold of audibility. The solid line 
in Fig. 7 represents a frequency distribution ofthe audio 
signal DA, and the bold tine represents the threshold of 

40 audibility of silent time. 

[0033] In psychological auditory sense encoding, in 
the first place, from the curves of the original sound and 
the threshold of audibility, the masking threshold (the 
broken line in Fig. 7) that is the boundary which can be 

45 perceived during the original sound is actually heard is 
calculated. Next, for each subband, the quantization 
step of each subband is allocated, in such a manner that 
the quantization noise becomes the smaller level than 
the masking threshold. 

so [0034] In Fig. 7, the ranges that are sampled by the 
compression processing are shown with the rectangle 
areas, and the bases of the rectangles correspond to 
the quantization noise levels. In such a region that the 
original sound is smaller than the masking threshold, the 

55 original sound is inaudible, and so it can be eliminated 
from the sample; on the other hand, in such a region 
that the sensitivity of auditory sense is high and the 
masking do not work sufficiently, the quantization steps 
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are made fine, and the level of the quantization noise is 
lowered hereby. 

[0035] Therefore, the psychological auditory sense 
analysis division 11 of the watermark encoder 1 analyz- 
es the digital audio signal DA1 based on the character- 
istics of psychological auditory sense encoding of the 
MP3 encoder 3 with respect to the inputted digital audio 
signal DA1, and, on the basis of the result of the very 
analysis, calculates the frequency band and its sound 
pressure in which the robust watermark DC1 and the 
fragile watermark DC2 are to be embedded, and con- 
trols the watermark superimposing division 16 by the re- 
sult of the very calculation (the psychological auditory 
sense encoded information D11). 
[0036] As a result of this, as shown in Fig. 8, by em- 
bedding a watermark in a rectangular region, which is 
sampled in compression processing, the very water- 
mark would function as the robust watermark DC1 that 
is resistant to compression, and if a watermark has been 
embedded in the outside of a rectangular region, which 
is eliminated through compression processing, the very 
watermark would function as the fragile watermark DC2 
that is easily affected by compression. 
[0037] On the basis of the psychological auditory 
sense encoded information D11 that has been calculat- 
ed by the psychological auditory sense analysis division 
11, the watermark superimposing division 16 embeds 
the robust watermark DC1 and the fragile watermark 
DC2 in the digital audio signal DA1 . 
[0038] In this connection, the region in which the wa- 
termark is embedded is not limited to a frequency re- 
gion, but may be a time region. 
[0039] Then, a MDCT coefficient D16, which is out- 
putted from the watermark superimposing division 1 6, 
is exposed to inverse orthogonal transformation in an 
I MDCT processing division 1 5, in this way it is outputted 
as the digital audio signal DA2 that is having the same 
form as the form of the original digital audio signal DA1 
and that the robust watermark DC1 and the fragile wa- 
termark DC2 have been embedded in. 
[0040] In this embodiment, the digital audio signal 
DA2 is encoded by the stated encoder (not shown), and 
used to expose an original master of a disk, and then 
an optical disk 2 is produced in volume on the basis of 
this original master disk. 

[0041 ] Fig. 9 shows a procedure of the watermark en- 
coder 1 , which processes the digital audio signal DA1 ; 
the watermark encoder 1 moves from the step SP11 to 
the step SP12, and sequentially reads the digital audio 
signal DA1 for each sample block. Then, the watermark 
encoder 1 goes to the step SP13, and analyzes the dig- 
ital audio signal DA1 , and extracts the psychological au- 
ditory sense encoded information D11 from the digital 
audio signal DA1 . 

[0042] Then, on the following step SP14, the water- 
mark encoder 1 embeds the robust watermark DC1 in 
the digital audio signal DA1 , on the basis of the psycho- 
logical auditory sense encoded information D11 ; in ad- 



dition, on the step SP1 5, it embeds the fragile watermark 
DC2 in the digital audio signal DA1 , on the basis of the 
psychological auditory sense encoded information D11 . 
[0043] In this way, the watermark encoder 1 obtains 
5 the digital audio signal DA2 that is having the same form 
as the form of the inputted digital audio signal DA1 and 
that the robust watermark DC1 and the fragile water- 
mark DC2 have been embedded in. 
[0044] The watermark encoder 1 then goes to the step 
10 SP1 6, and judges whether the processing of the digital 
audio signal DA1 has been completed or not; when the 
negative result has been obtained here, it returns to the 
step SP 12. In this way, the watermark encoder 1 repeats 
this procedure for each sample block in sequence, so 
'5 as to process the digital audio signal DA1 ; when the af- 
firmative result has been obtained on the step SP16, it 
moves from the step SP1 6 to the step SP1 7, and ends 
the very procedure. 

[0045] Fig. 1 0 shows a configuration of the watermark 
decoder 6 of the personal-computer 5, which has been 
stated above with reference to Fig. 1 ; the watermark de- 
coder 6 detects the robust watermark DC1 and the frag- 
ile watermark DC2 from the digital audio signal DA2, 
which is obtained by a playback of the optical disk 2 (Fig. 
1). 

[0046] That is, the watermark decoder 6 performs 
MDCT processing of the sequentially inputted digital au- 
dio signal DA2 in a MDCT processing division 60, and 
obtains a MDCT coefficient D60 hereby, and then out- 
puts this to the watermark detecting division 61 . 
[0047] With respect to the inputted MDCT coefficient 
D60, the watermark detecting division 61 shifts each of 
plural frequency components that have been previously 
and separately allocated to the robust watermark DC1 
and the fragile watermark DC2 as their embedding fre- 
quency bands by, for instance, 4 f req uency components 
in such a direction that the frequencies increase; on the 
basis of the new MDCT coefficient that has been ob- 
tained by the very shifting process and the MDCT coef- 
ficient D60 that is obtained from the MDCT processing 
division 60, it compares the polarities of the frequency 
components, and detects the robust watermark DC1 
and the fragile watermark DC2 on the basis the lean of 
the polarity. 

[0048] In this connection, the region in which the wa- 
termark is detected is not limited to a frequency region, 
but may be a time region. 

[0049] When only the robust watermark DC1 has 
been detected at the time of copying of the digital audio 
signal DA2 that has been recorded on the optical disk 
2, this is representing that there is a history of compres- 
sion based on MP3, that is, the digital audio signal DA2 
in which the robust watermark DC1 that has been de- 
tected at this time has been embedded is the contents 
that has been distributed via the network 4; so, when 
copying restriction is previously set by the copyrighter, 
by restricting the copying, it is possible to prevent illegal 
copying that constitutes an infringement of the copy- 
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right. 

[0050] Besides, when both of the robust watermark 
DC1 and the fragile watermark DC2 have been detect- 
ed, this is representing that there is not any history of 
compression based on MP3, that is, it is not what has 
been distributed via the network 4, and that the copy- 
righter releases it free of charge; so, it can be known 
that the copying restriction is needless. 
[0051] Thus, in the personal computer 5, it is possible 
to restrict the copying, on the basis of the copyright in- 
formation and the compression history information 
based on the robust watermark DC1 and the fragile wa- 
termark DC2. 

[0052] Fig. 1 1 shows procedure of the watermark de- 
coder 6 for processing the digital audio signal DA2; the 
watermark decoder 6 enters the very procedure at the 
step SP21 and goes to the step SP22 : and reads the 
digital audio signal DA2 in sequence for every sample 
block. Then, the watermark decoder 6 goes to the step 
SP23 and detects the robust watermark DC1 from the 
digital audio signal DA2, and detects the fragile water- 
mark DC2 at the following step S P24. 
[0053] After the detection of the fragile watermark 
DC2, the watermark decoder 6 goes to the step SP25 
and judges whether the processing of the digital audio 
signal DA2 has been completed or not; when the nega- 
tive result has been obtained, it returns to the step SP22. 
In this way, the watermark decoder 6 repeats this pro- 
cedure in sequence for each sample block, so as to 
process the digital audio signal DA2; when the affirma- 
tive result has been obtained at the step SP25 : it goes 
to the step SP26 and ends the very procedure. 
[0054] In the above configuration, the psychological 
auditory sense analysis division 1 1 of the watermark en- 
coder 1 determines the embedding position (for in- 
stance, frequency band) and the sound pressure level, 
in response to the compression characteristics of the 
MP3 encoder 3 (psychological auditory sense models 
of threshold of audibility, masking effect and so on that 
have been adopted in the various compression stand- 
ards), so that it is possible to embed the fragile water- 
mark DC2 that is easily affected by compression and 
that meets the compression characteristics of the MP3 
encoder 3 in the digital audio signal DA1 . 
[0055] Therefore, when the fragile watermark DC2 
has been compressed via the MP3 encoder 3 : its sur- 
vival rate would be lowered sufficiently, and it would not 
be detected at the watermark decoder 6. 
[0056] In this way, according to the above configura- 
tion, it is possible to easily realize the robust watermark 
DC1 that is resistant to compression and the fragile wa- 
termark DC2 that is affected by compression. In this 
connection, by embedding copyright information in the 
robust watermark DC1 , which has strong compression 
resistance, it is possible to render the copyright informa- 
tion difficult to be detected on the basis of the robust 
watermark DC1 that has been detected after it has 
passed through the compression system. Moreover, the 



compression history can be determined on the basis of 
the reduction or elimination of the fragile watermark DC2 
after it has passed through the compression system so 
that the illegal copying on the network can be prevented. 

5 [0057] In the above embodiment, such a case has 
been described that the robust watermark DC1 and the 
fragile watermark DC2 are detected from the digital au- 
dio signal that is obtained by a playback of the optical 
disk 2; however, we do not intend to limit the present 

io invention to such. The present invention is applicable to 
also such a case that the watermark is detected from a 
stream signal in which the digital audio signal has been 
compress-processed. If do it like this, it is possible to 
eliminate trouble of extending the compressed stream 

is signal and transforming it to the same form as that of 
the original digital audio signal, and it is possible to en- 
hance the speed of the processing. 
[0058] When embedding the robust watermark- DC 1 
and the fragile watermark DC2 in the digital audio signal, 

20 the fragile watermark DC2 may be embedded in a form 
that it may not be deleted by predetermined compres- 
sion processing. 

[0059] In this case, the fragile watermark DC2 may be 
embedded in a form that it may not be deleted by proper 
25 compression processing performed at the time of distri- 
bution, and also embedded in a form that it may be de- 
leted when the digital audio signal with a watermark is 
compressed by other types of compression processing 
such as MP3. More concretely, compression character- 
30 istic of the proper compression processing used at the 
time of distribution and that of other types of compres- 
sion processing are compared. The proper compression 
processing used at the time of distribution does not de- 
lete the fragile watermark DC2. The other types of com- 
35 pression processing embeds the fragile watermark DC2 
into signals that are deleted by quantization processing 
for the digital audio signal, based on the above-men- 
tioned threshold of audibility and masking threshold at 
the time of compression processing. 
40 [0060] Furthermore, in the above embodiment, such 
a case has been described that the robust watermark 
DC1 and the fragile watermark DC2 are embedded in 
the digital audio signal; however, we do not intend to 
limit the present invention to such. The robust water- 
45 mark DC1 is embedded in the digital audio signal, and 
after the compression processing of the signal, the frag- 
ile watermark DC2 may be embedded in a compressed 
stream signal. As a result, it is possible to easily set the 
deletion level of the fragile watermark DC2 by compres- 
50 sion processing to the minimum, regardless of the char- 
acteristic of the Codec. 

[0061] Moreover, in the above embodiment, such a 
case has been described that the fragile watermark DC2 
is embedded in the digital audio signal at the position 
55 and level to be completely deleted after compression 
processing such as MP3; however, we do not intend to 
limit the present invention to such. The fragile water- 
mark DC2 may not be completely deleted even after the 
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compression processing such as MP3. 
[0062] The detection division determines whether or 
not the detected level of the fragile watermark DC2 is 
lower than a predetermined threshold. If the level is be- 
low the threshold, it may be set to determine that the 
digital audio signal to be detected has been illegally cop- 
ied. If beyond the threshold, it may be set to determine 
that the digital audio signal to be detected has been 
properly copied. After all, the fragile watermark DC2 
may be embedded in the digital audio signal in any com- 
position, as long as the detection division can detect the 
fact that the compression processing has been per- 
formed according to how low the survival rate of the frag- 
ile watermark DC2 has become after the digital audio 
signal embedded with the robust watermark DC1 and 
the fragile watermark DC2 is compressed such as by 
MP3. 

[0063] Besides, in the above embodiment, such a 
case has been described that the robust watermark DC1 
and the fragile watermark DC2 are embedded in the dig- 
ital audio signal, using the psychological auditory sense 
model that is the compression characteristics of the 
MP3; however, we do not intend to limit the present in- 
vention to such. The present invention can be widely ap- 
plied to such cases that psychological auditory sense 
models based on a wide variety of compression stand- 
ards such as MPEG-AAC, ATRAC, ATRAC2, ATRAC3, 
Dolby AC3, MS (Microsoft) Audio, Twin VQ, or combi- 
nations of these psychological auditory sense models 
are used. If do it like this, it is possible to realize a gen- 
eral-use fragile watermark that suits much compression 
standards. 

[0064] Besides, in the above embodiment, such a 
configuration has been described that the robust water- 
mark encoding division has been integrated with the 
fragile watermark encoding division in the watermark 
encoder 1 ; however, we do not intend to limit the present 
invention to such. For instance, the present invention is 
able to use such a configuration that the robust water- 
mark encoding division is connected in series with the 
fragile watermark encoding division, or such a configu- 
ration that the robust watermark encoding division is 
connected in a parallel arrangement with the fragile wa- 
termark encoding division. 

[0065] Besides, in the above embodiment, such a 
configuration has been described that the robust water- 
mark decoding division has been integrated with the 
fragile watermark decoding division in the watermark 
decoder 6; however, we do not intend to limit the present 
invention to such. For instance, the present invention is 
able to use such a configuration that the robust water- 
mark decoding division is connected in series with the 
fragile watermark decoding division, or such a configu- 
ration that the robust watermark decoding division is 
connected in a parallel arrangement with the fragile wa- 
termark decoding division. 

[0066] Besides, in the above embodiment, such a 
case has been described that the copyright information 



and the compression history information is superim- 
posed on the digital audio signal; however, we do not 
intend to limit the present invention to such. The present 
invention may be applied to the cases where the various 
5 information is superimposed on, as necessary, and 
transferred. 

[0067] Besides, in the above embodiment, such a 
case has been described that the embedding process 
of the watermark is performed by the watermark encod- 
10 er 1 ; however, we do not intend to limit the present in- 
vention to such. Alternatively, from a program storing 
medium (a floppy disk, an optical disk and so on) in 
which a program for realizing the respective functions 
of the water mark encoder 1 has been stored, the pro- 
's gram may be loaded in an information processing de- 
vice (computer), and the respective functions may be 
executed. 

[0068] For example, an ordinary computer 70 as 
shown in Fig. 12 comprises a CPU (Central Processing 
20 Unit) 71 in charge of an entire operation, a ROM (Read 
Only Memory) 72 storing various kinds of software, a 
RAM (Random Access Memory) 73 as a work memory 
for the CPU 71 , a hard disk drive 74 storing various data, 
and a communication port 75 which is an interface for 
25 communicating with the externals via a network. These 
are interconnected though a bus 76 to make a configu- 
ration. 

[0069] The hard disk drive 74 or the ROM 72 stores 
programs to perform each function mentioned above in 
30 the computer 70. At the time of execution, the CPU 71 
may activate each corresponding function part accord- 
ing to the work memory stored in the RAM 73. Also, the 
computer 70 may install programs for activating each 
function mentioned above via the communication port 
35 75. 

Industrial Applicability 

[0070] This invention is directed to a signal process- 
40 ing device and its method as well as a program storing 
medium, and applicable to the safeguarding of the cop- 
yright of music data and so on. 



1 . A signal processing device for superimposing digit- 
al watermarking information on the given input sig- 
nal, said device comprises: 

50 

psychological auditory sense analysis means 
for performing psychological auditory sense 
analysis of the survival state of said input signal 
of the time when said input signal has been 
55 compressed, in response to said compression 

processing, and for outputting the result of the 
very analysis as the psychological auditory 
sense encoded information; and 
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superimposing means for superimposing said 
digital watermarking information on said input 
signal, on the basis of said psychological audi- 
tory sense encoded information. 

5 

The signal processing device according to claim 1 , 
wherein 

said psychological auditory sense analysis 
means generates said psychological auditory 
sense encoded information, on the basis of the n> 
threshold of audibility. 

The signal processing device according to claim 1 , 
wherein 

said psychological auditory sense analysis 15 
means generates said psychological auditory 
sense encoded information, on the basis of the 
masking effect of the audio signal. 

The signal processing device according to claim 1 , 20 
wherein 

said superimposing means superimpose dig- 
ital watermarking information that has large com- 
pression resistance toward said compression 
processing and digital watermarking information 25 
that has small compression resistance toward said 
compression processing on said input signal, on the 
basis of said psychological auditory sense encoded 
information. 

30 

The signal processing device according to claim 1 , 
wherein 

said digital watermarking information is the 
copyright information relevant to said input signal. 

35 

A signal processing method of superimposing dig- 
ital watermarking information on the given input sig- 
nal, said method comprises: 

a step of performing psychological auditory 40 
sense analysis of the survival state of said input 
signal of the time when said input signal has 
been compressed, in response to said com- 
pression processing, and then outputting the 
result of the very analysis as the psychological 45 
auditory sense encoded information; and 
a step of superimposing said digital watermark- 
ing information on said input signal, on the ba- 
sis of said psychological auditory sense encod- 
ed information. so 

The signal processing method according to claim 6, 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- ss 
formation is generated, on the basis of the threshold 
of audibility. 



8. The signal processing method according to claim 6, 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
formation is generated, on the basis of the masking 
effect of the audio signal. 

9. The signal processing method according to claim 6, 
wherein 

at said step of superimposing said digital wa- 
termarking information on said input signal, digital 
watermarking information that has large compres- 
sion resistance toward said compression process- 
ing and digital watermarking information that has 
small compression resistance toward said com- 
pression processing are superimposed on said in- 
put signal, on the basis of said psychological audi- 
tory sense encoded information. 

1 0. The signal processing method according to claim 6, 
wherein 

said digital watermarking information is the 
copyright information relevant to said input signal. 

11. A program storing medium for causing a signal 
processing device to execute a program compris- 
ing: 

a step of performing psychological auditory 
sense analysis of the survival state of said input 
signal of the time when said input signal has 
been compressed, in response to said com- 
pression processing, and then outputting the 
result of the very analysis as the psychological 
auditory sense encoded information; and 
a step of superimposing said digital watermark- 
ing information on said input signal, on the ba- 
sis of said psychological auditory sense encod- 
ed information. 

12. The program storing medium according to claim 11, 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
formation is generated, on the basis of the threshold 
of audibility. 

1 3. The program storing medium according to claim 11 , 
wherein 

at said psychological auditory sense analysis 
step, said psychological auditory sense encoded in- 
formation is generated, on the basis of the masking 
effect of the audio signal. 

14. The program storing medium according to claim 11 , 
wherein 

at said step of superimposing said digital wa- 
termarking information on said input signal, digital 
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watermarking Information that has large compres- 
sion resistance toward said compression process- 
ing and digital watermarking information that has 
small compression resistance toward said com- 
pression processing are superimposed on said in- 5 
put signal, on the basis of said psychological audi- 
tory sense encoded information. 

1 5. The program storing medium according to claim 1 1 , 
wherein 10 

said digital watermarking information is the 
copyright information relevant to said input signal. 
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